18/08/2024 - 24/08/2024

19/08/2024 16:03

The entry here:

13/08/2024 23:29

Tried making a new midas install on the SSD and linking it to dependencies already on the HDD. The idea was to somehow remove any knowledge of the HDD from midas. It didn't seem to work, supporting the theory that the bottlneck lies in mlogger instead.

5kHz with waveform size 4800

5kHz with waveform size 7200:

The conclusion is will still max at around 240 MB/s

was actually using an installation of midas on the HDD, so I'm retrying with an installation of midas on the SSD.

19/08/2024 17:24

After ensuring it now uses a midas version on the SSD, I find the same results.

19/08/2024 18:16

After playing with the Data Simulator, it appears midas has some problems when the bank size gets too large:

INT read_periodic_event(char *pevent, INT off)
{
   short *pdata; // Change the data type to short

   // Init bank structure
   bk_init32(pevent);

   // Create a bank named "CR00" and specify the data type as TID_SHORT
   bk_create(pevent, "CR00", TID_SHORT, (void **)&pdata);

    // Repeat the loop to scale the data
    for (int repeat = 0; repeat < 400; repeat++) {
        for (int i = 0; i < data.size(); i++) {
            *pdata++ = data[i];
        }
    }

   // Close the bank
   bk_close(pevent, pdata);

   return bk_size(pevent);
}

INT read_periodic_event(char *pevent, INT off)
{
   short *pdata; // Change the data type to short

   // Init bank structure
   bk_init32(pevent);

   // Create a bank named "CR00" and specify the data type as TID_SHORT
   bk_create(pevent, "CR00", TID_SHORT, (void **)&pdata);

    // Repeat the loop to scale the data
    for (int repeat = 0; repeat < 400; repeat++) {
        for (int i = 0; i < data.size(); i++) {
            *pdata++ = data[i];
        }
    }

   // Close the bank
   bk_close(pevent, pdata);

   return bk_size(pevent);
}

Each data vector here is about 1kB. Midas complains about null pointers when I try scaling by 1000 for each event (so ~1MB/event) at 100Hz. When I try scaling by 500 for each event at 100Hz I get the following error messages:

18:11:17.086 2024/08/19 [Logger,ERROR] [odb.cxx:5497:db_get_data_locked,ERROR] odb entry "/Equipment/Data Simulator/Variables/CR00" data truncated, size is 113600 (56800*2), buffer size is only 0

18:11:16.987 2024/08/19 [DataSimulator,ERROR] [midas.cxx:17635:cm_write_event_to_odb,ERROR] cannot write bank "CR00" to ODB, db_set_data1() status 310

18:11:16.986 2024/08/19 [DataSimulator,ERROR] [odb.cxx:6999:db_set_data1,ERROR] Cannot reallocate "/Equipment/Data Simulator/Variables/CR00" with new size 568000 bytes, online database full

18:11:16.986 2024/08/19 [DataSimulator,ERROR] [odb.cxx:559:realloc_data,ERROR] cannot malloc_data(568000), called from db_set_data1

18:11:17.086 2024/08/19 [Logger,ERROR] [odb.cxx:5497:db_get_data_locked,ERROR] odb entry "/Equipment/Data Simulator/Variables/CR00" data truncated, size is 113600 (56800*2), buffer size is only 0

18:11:16.987 2024/08/19 [DataSimulator,ERROR] [midas.cxx:17635:cm_write_event_to_odb,ERROR] cannot write bank "CR00" to ODB, db_set_data1() status 310

18:11:16.986 2024/08/19 [DataSimulator,ERROR] [odb.cxx:6999:db_set_data1,ERROR] Cannot reallocate "/Equipment/Data Simulator/Variables/CR00" with new size 568000 bytes, online database full

18:11:16.986 2024/08/19 [DataSimulator,ERROR] [odb.cxx:559:realloc_data,ERROR] cannot malloc_data(568000), called from db_set_data1

19/08/2024 18:59

I made some adjustments and was able to write at 1GB/s on the SSD. In particular I added the features:

Use a polling system to trigger events at a given rate
Each event is a scalable vector of zeros
Events are memcpyed into the data bank

This indicates mlogger (or otherwise midas) is NOT the bottleneck. Interestingly enough, this also is faster than the tested write speed for the drive using:

/********************************************************************\

  Name:         wdfe.cxx
  Created by:   Stefan Ritt

  Contents:     Example front-end for standalone WaveDREAM board

\********************************************************************/

#include <stdio.h>
#include <stdlib.h>
#include <math.h>
#include <string.h>
#include <iostream>
#include <fstream>
#include <sstream>
#include <vector>
#include "midas.h"
#include "mfe.h"
#include <stdlib.h> // Include the header for rand()
#include <random> // Include for random number generation


void trigger_update(INT, INT, void*);

/*-- Globals -------------------------------------------------------*/

/* The frontend name (client name) as seen by other MIDAS clients   */
const char *frontend_name = "DataSimulator";
/* The frontend file name, don't change it */
const char *frontend_file_name = __FILE__;

/* frontend_loop is called periodically if this variable is TRUE    */
BOOL frontend_call_loop = FALSE;

/* a frontend status page is displayed with this frequency in ms */
INT display_period = 1000;

/* maximum event size produced by this frontend */
INT max_event_size = 1024 * 1014;

/* maximum event size for fragmented events (EQ_FRAGMENTED) */
INT max_event_size_frag = 5 * max_event_size;

/* buffer size to hold events */
INT event_buffer_size = 5 * max_event_size;

// Define a vector to store 16-bit words
std::vector<int16_t> data; // Define a global vector to store 16-bit signed integers

// Global variable to keep track of the last poll time
std::chrono::steady_clock::time_point last_poll_time;
const std::chrono::microseconds polling_interval(100); // Poll every 100 microsecond

// Random number generator for generating data
std::mt19937 generator;
std::uniform_int_distribution<short> distribution(-32768, 32767); // Define the range of random values (short range)

// Global variable to hold the zero buffer
std::vector<short> zero_buffer;


/*-- Function declarations -----------------------------------------*/

INT frontend_init(void);
INT frontend_exit(void);
INT begin_of_run(INT run_number, char *error);
INT end_of_run(INT run_number, char *error);
INT pause_run(INT run_number, char *error);
INT resume_run(INT run_number, char *error);
INT frontend_loop(void);

INT read_trigger_event(char *pevent, INT off);
INT read_periodic_event(char *pevent, INT off);

INT poll_event(INT source, INT count, BOOL test);
INT interrupt_configure(INT cmd, INT source, POINTER_T adr);

/*-- Equipment list ------------------------------------------------*/

BOOL equipment_common_overwrite = TRUE;

EQUIPMENT equipment[] = {
   {"Data Simulator",              /* equipment name */
      {2, 0,                 /* event ID, trigger mask */
         "SYSTEM",           /* event buffer */
         EQ_POLLED,        /* equipment type */
         0,                  /* event source */
         "MIDAS",            /* format */
         TRUE,               /* enabled */
         RO_RUNNING | RO_TRANSITIONS |   /* read when running and on transitions */
         RO_ODB,             /* and update ODB */
         10,               /* read every sec */
         0,                  /* stop run after this event limit */
         0,                  /* number of sub events */
         TRUE,               /* log history */
         "", "", "",},
      read_trigger_event   /* readout routine */
   },

   {""}
};

/*-- Trigger Update ------------------------------------------------*/

void trigger_update(INT hDB, INT hkey,void*)
{

}


/*-- Frontend Init -------------------------------------------------*/

int frontend_init() {
    // Open the file for reading
    std::ifstream inputFile("fake_data.txt");

    if (!inputFile) {
        std::cerr << "Error opening the file." << std::endl;
        return 1;
    }

    std::cout << "Reading and converting data:" << std::endl;

    std::string line;
    while (std::getline(inputFile, line)) {
        std::istringstream iss(line);
        std::string token;

        while (std::getline(iss, token, ',')) {
            int16_t value;
            std::istringstream(token) >> value;
            data.push_back(value);
        }
    }

    // Print the converted data
    for (int i = 0; i < data.size(); i++) {
        std::cout << " " << data[i];
    }

    // Close the file
    inputFile.close();

    if (data.empty()) {
        std::cerr << "No data was converted." << std::endl;
    } else {
        std::cout << std::endl << "Conversion completed." << std::endl;
    }

    // Initialize random number generator
    std::random_device rd; // Obtain a random number from hardware
    generator = std::mt19937(rd()); // Seed the generator

    // Define the total number of zero data points
    const int total_data_size = 50000; // Adjust size as needed

    // Create and initialize the buffer of zeros
    zero_buffer.resize(total_data_size, 0);

    return SUCCESS;
}




/*-- Frontend Exit -------------------------------------------------*/

INT frontend_exit()
{
   return SUCCESS;
}

/*-- Begin of Run --------------------------------------------------*/

INT begin_of_run(INT run_number, char *error)
{
   return SUCCESS;
}

/*-- End of Run ----------------------------------------------------*/

INT end_of_run(INT run_number, char *error)
{
   return SUCCESS;
}

/*-- Pause Run -----------------------------------------------------*/

INT pause_run(INT run_number, char *error)
{
   return SUCCESS;
}

/*-- Resume Run ----------------------------------------------------*/

INT resume_run(INT run_number, char *error)
{
   return SUCCESS;
}

/*-- Frontend Loop -------------------------------------------------*/

INT frontend_loop()
{
   /* if frontend_call_loop is true, this routine gets called when
      the frontend is idle or once between every event */
   return SUCCESS;
}

/*------------------------------------------------------------------*/

/********************************************************************\

  Readout routines for different events

\********************************************************************/

/*-- Trigger event routines ----------------------------------------*/

INT poll_event(INT source, INT count, BOOL test) {
    // Get the current time
    auto now = std::chrono::steady_clock::now();
    
    // Check if enough time has passed since the last poll
    if (now - last_poll_time >= polling_interval) {
        // Update the last poll time
        last_poll_time = now;
        
        // Return TRUE to indicate that an event is available
        return TRUE;
    }
    
    // If test is TRUE, don't return anything
    if (test) {
        return FALSE;
    }
    
    // Otherwise, return FALSE to indicate no event available
    return FALSE;
}

/*-- Interrupt configuration ---------------------------------------*/

INT interrupt_configure(INT cmd, INT source, POINTER_T adr)
{
   switch (cmd) {
   case CMD_INTERRUPT_ENABLE:
      break;
   case CMD_INTERRUPT_DISABLE:
      break;
   case CMD_INTERRUPT_ATTACH:
      break;
   case CMD_INTERRUPT_DETACH:
      break;
   }
   return SUCCESS;
}

/*-- Event readout -------------------------------------------------*/

INT read_trigger_event(char *pevent, INT off)
{
    short *pdata;

    // Init bank structure
    bk_init32(pevent);

    // Create a bank named "CR00" and specify the data type as TID_SHORT
    bk_create(pevent, "CR00", TID_SHORT, (void **)&pdata);

    // Use memcpy to copy the buffer of zeros into the MIDAS bank
    memcpy(pdata, zero_buffer.data(), zero_buffer.size() * sizeof(short));

    // Adjust pdata pointer
    pdata += zero_buffer.size();  // Move the pointer past the copied data

    // Close the bank
    bk_close(pevent, pdata);

    return bk_size(pevent);
}

/*-- Periodic event ------------------------------------------------*/

INT read_periodic_event(char *pevent, INT off)
{
   short *pdata; // Change the data type to short

   // Init bank structure
   bk_init32(pevent);

   // Create a bank named "CR00" and specify the data type as TID_SHORT
   bk_create(pevent, "CR00", TID_SHORT, (void **)&pdata);

    // Repeat the loop 5000 times
    for (int repeat = 0; repeat < 400; repeat++) {
        for (int i = 0; i < data.size(); i++) {
            *pdata++ = data[i];
        }
    }

   // Close the bank
   bk_close(pevent, pdata);

   return bk_size(pevent);
}

/********************************************************************\

  Name:         wdfe.cxx
  Created by:   Stefan Ritt

  Contents:     Example front-end for standalone WaveDREAM board

\********************************************************************/

#include <stdio.h>
#include <stdlib.h>
#include <math.h>
#include <string.h>
#include <iostream>
#include <fstream>
#include <sstream>
#include <vector>
#include "midas.h"
#include "mfe.h"
#include <stdlib.h> // Include the header for rand()
#include <random> // Include for random number generation


void trigger_update(INT, INT, void*);

/*-- Globals -------------------------------------------------------*/

/* The frontend name (client name) as seen by other MIDAS clients   */
const char *frontend_name = "DataSimulator";
/* The frontend file name, don't change it */
const char *frontend_file_name = __FILE__;

/* frontend_loop is called periodically if this variable is TRUE    */
BOOL frontend_call_loop = FALSE;

/* a frontend status page is displayed with this frequency in ms */
INT display_period = 1000;

/* maximum event size produced by this frontend */
INT max_event_size = 1024 * 1014;

/* maximum event size for fragmented events (EQ_FRAGMENTED) */
INT max_event_size_frag = 5 * max_event_size;

/* buffer size to hold events */
INT event_buffer_size = 5 * max_event_size;

// Define a vector to store 16-bit words
std::vector<int16_t> data; // Define a global vector to store 16-bit signed integers

// Global variable to keep track of the last poll time
std::chrono::steady_clock::time_point last_poll_time;
const std::chrono::microseconds polling_interval(100); // Poll every 100 microsecond

// Random number generator for generating data
std::mt19937 generator;
std::uniform_int_distribution<short> distribution(-32768, 32767); // Define the range of random values (short range)

// Global variable to hold the zero buffer
std::vector<short> zero_buffer;


/*-- Function declarations -----------------------------------------*/

INT frontend_init(void);
INT frontend_exit(void);
INT begin_of_run(INT run_number, char *error);
INT end_of_run(INT run_number, char *error);
INT pause_run(INT run_number, char *error);
INT resume_run(INT run_number, char *error);
INT frontend_loop(void);

INT read_trigger_event(char *pevent, INT off);
INT read_periodic_event(char *pevent, INT off);

INT poll_event(INT source, INT count, BOOL test);
INT interrupt_configure(INT cmd, INT source, POINTER_T adr);

/*-- Equipment list ------------------------------------------------*/

BOOL equipment_common_overwrite = TRUE;

EQUIPMENT equipment[] = {
   {"Data Simulator",              /* equipment name */
      {2, 0,                 /* event ID, trigger mask */
         "SYSTEM",           /* event buffer */
         EQ_POLLED,        /* equipment type */
         0,                  /* event source */
         "MIDAS",            /* format */
         TRUE,               /* enabled */
         RO_RUNNING | RO_TRANSITIONS |   /* read when running and on transitions */
         RO_ODB,             /* and update ODB */
         10,               /* read every sec */
         0,                  /* stop run after this event limit */
         0,                  /* number of sub events */
         TRUE,               /* log history */
         "", "", "",},
      read_trigger_event   /* readout routine */
   },

   {""}
};

/*-- Trigger Update ------------------------------------------------*/

void trigger_update(INT hDB, INT hkey,void*)
{

}


/*-- Frontend Init -------------------------------------------------*/

int frontend_init() {
    // Open the file for reading
    std::ifstream inputFile("fake_data.txt");

    if (!inputFile) {
        std::cerr << "Error opening the file." << std::endl;
        return 1;
    }

    std::cout << "Reading and converting data:" << std::endl;

    std::string line;
    while (std::getline(inputFile, line)) {
        std::istringstream iss(line);
        std::string token;

        while (std::getline(iss, token, ',')) {
            int16_t value;
            std::istringstream(token) >> value;
            data.push_back(value);
        }
    }

    // Print the converted data
    for (int i = 0; i < data.size(); i++) {
        std::cout << " " << data[i];
    }

    // Close the file
    inputFile.close();

    if (data.empty()) {
        std::cerr << "No data was converted." << std::endl;
    } else {
        std::cout << std::endl << "Conversion completed." << std::endl;
    }

    // Initialize random number generator
    std::random_device rd; // Obtain a random number from hardware
    generator = std::mt19937(rd()); // Seed the generator

    // Define the total number of zero data points
    const int total_data_size = 50000; // Adjust size as needed

    // Create and initialize the buffer of zeros
    zero_buffer.resize(total_data_size, 0);

    return SUCCESS;
}




/*-- Frontend Exit -------------------------------------------------*/

INT frontend_exit()
{
   return SUCCESS;
}

/*-- Begin of Run --------------------------------------------------*/

INT begin_of_run(INT run_number, char *error)
{
   return SUCCESS;
}

/*-- End of Run ----------------------------------------------------*/

INT end_of_run(INT run_number, char *error)
{
   return SUCCESS;
}

/*-- Pause Run -----------------------------------------------------*/

INT pause_run(INT run_number, char *error)
{
   return SUCCESS;
}

/*-- Resume Run ----------------------------------------------------*/

INT resume_run(INT run_number, char *error)
{
   return SUCCESS;
}

/*-- Frontend Loop -------------------------------------------------*/

INT frontend_loop()
{
   /* if frontend_call_loop is true, this routine gets called when
      the frontend is idle or once between every event */
   return SUCCESS;
}

/*------------------------------------------------------------------*/

/********************************************************************\

  Readout routines for different events

\********************************************************************/

/*-- Trigger event routines ----------------------------------------*/

INT poll_event(INT source, INT count, BOOL test) {
    // Get the current time
    auto now = std::chrono::steady_clock::now();
    
    // Check if enough time has passed since the last poll
    if (now - last_poll_time >= polling_interval) {
        // Update the last poll time
        last_poll_time = now;
        
        // Return TRUE to indicate that an event is available
        return TRUE;
    }
    
    // If test is TRUE, don't return anything
    if (test) {
        return FALSE;
    }
    
    // Otherwise, return FALSE to indicate no event available
    return FALSE;
}

/*-- Interrupt configuration ---------------------------------------*/

INT interrupt_configure(INT cmd, INT source, POINTER_T adr)
{
   switch (cmd) {
   case CMD_INTERRUPT_ENABLE:
      break;
   case CMD_INTERRUPT_DISABLE:
      break;
   case CMD_INTERRUPT_ATTACH:
      break;
   case CMD_INTERRUPT_DETACH:
      break;
   }
   return SUCCESS;
}

/*-- Event readout -------------------------------------------------*/

INT read_trigger_event(char *pevent, INT off)
{
    short *pdata;

    // Init bank structure
    bk_init32(pevent);

    // Create a bank named "CR00" and specify the data type as TID_SHORT
    bk_create(pevent, "CR00", TID_SHORT, (void **)&pdata);

    // Use memcpy to copy the buffer of zeros into the MIDAS bank
    memcpy(pdata, zero_buffer.data(), zero_buffer.size() * sizeof(short));

    // Adjust pdata pointer
    pdata += zero_buffer.size();  // Move the pointer past the copied data

    // Close the bank
    bk_close(pevent, pdata);

    return bk_size(pevent);
}

/*-- Periodic event ------------------------------------------------*/

INT read_periodic_event(char *pevent, INT off)
{
   short *pdata; // Change the data type to short

   // Init bank structure
   bk_init32(pevent);

   // Create a bank named "CR00" and specify the data type as TID_SHORT
   bk_create(pevent, "CR00", TID_SHORT, (void **)&pdata);

    // Repeat the loop 5000 times
    for (int repeat = 0; repeat < 400; repeat++) {
        for (int i = 0; i < data.size(); i++) {
            *pdata++ = data[i];
        }
    }

   // Close the bank
   bk_close(pevent, pdata);

   return bk_size(pevent);
}

Interestingly, this is faster than what dd reports for the write speed of the drive

[root@dhcp-10-163-105-238 ~]# dd if=/dev/zero of=/data/ssd/testfile bs=1G count=1 oflag=dsync
 HDD
dd if=/dev/zero of=/home/testfile bs=1G count=1 oflag=dsync
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB) copied, 1.45859 s, 736 MB/s

[root@dhcp-10-163-105-238 ~]# dd if=/dev/zero of=/data/ssd/testfile bs=1G count=1 oflag=dsync
 HDD
dd if=/dev/zero of=/home/testfile bs=1G count=1 oflag=dsync
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB) copied, 1.45859 s, 736 MB/s

In both cases, we are just writing zeros to the drive. I don't know why this is (and don't care to investigate).

19/08/2024 19:09

By changing the logging directory from
/data/ssd/simdaq_data (SSD) to /home/installation_testing/online/ (HDD) I saw the data rate go down to ~110 MB/s from ~1GB/s (did not test past 1GB/s).

This is evidence that yes, the drive does matter. We were able to achieve 240MB/s (with 50% compression rate so 120MB/s) a with the HDD with the gm2daq which seems consistent with the 110MB/s I'm seeing here.

I tested lower event rates (~3kHz) and saw the same results.

Anyways, the overall conclusion is the bottleneck in the gm2daq is not the logger as it can handle 1GB/s. I'm still skeptical that somehow the HDD is causing a bottleneck in the gm2daq, even when we write to SSD.